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My  interest  in  understanding  fundamental  limits  and  practical  algorithms/codes  to  instantiate 
such  limits  -  within  the  context  of  communication  based  upon  packet  timings  -  was  inspired  by 
my  understanding  how  neurons  signal  to  one  another  with  the  timings  of  their  spikes. 
Remarkably,  using  this  modality  to  communicate  over  packet  networks  has  not  been  explored  in 
nearly  as  much  depth  as  other  communication  modalities.  Applications  of  this  approach  are 
abundant:  including  unequal  error  protection,  covert  communications,  and  computer  network 
security. 

Although  an  1990s  paper  titled  "Bits  through  queues”  characterized  fundamental  limits  of 
communication  with  packet  timings  over  queuing  channels,  my  curiosity  led  to  a  very  different  - 
and  in  some  sense  simpler  -  way  to  arrive  at  the  fundamental  limits  of  such  complicated 
queuing  channels  (that  are  nonlinear  and  have  memory).  By  considering  the  queuing  system  as 
a  stochastic  dynamical  system  with  feedback  coupled  to  a  memoryless  channel,  I  was  able  to 
provide  a  much  simpler  explanation  of  the  fundamental  limits  of  such  a  system  [1],  Also,  this 
new  proof  methodology  can  be  extended  to  many  other  information-theoretic  contexts,  which  I 
have  also  been  actively  pursuing  [2,3],  Lastly,  this  new  proof  technique  has  led  to  the 
development  of  low-complexity,  provably  good,  error-correcting  codes  that  can  be  used  over 
computer  networks  [4].  Extensions  of  this  approach  and  its  industrial  potential  have  led  to  us 
winning  UlUC’s  Grainger  Award  in  Emerging  Technologies  in  2008. 
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2008. 


My  curiosity  about  developing  canonical  design  principles  for  brain-computer  interfaces  has  led 
to  an  interpretation  of  the  brain-computer  interface  as  two  cooperating  agents  (the  brain  and 
computer)  with  different  pieces  of  information,  who  are  attempting  to  cooperate  with  one  another 
via  noisy  communication  channels  between  them.  The  user’s  intent  -  in  short  -  must  be 
communicated  to  the  interface  via  recorded  brain  signals  -  and  given  feedback  from  the 
interface’s  dynamics  -  so  that  it  may  help  the  user  accomplish  a  task.  This  new  interpretation, 
wed  with  recent  results  on  implementable,  optimal,  feedback  information  theoretic  coding 
schemes  (titled  “posterior  matching  schemes”),  led  us  to  design  brain-machine  interface 
paradigms  from  decentralized  control  and  feedback  information-theoretic  perspective  (see 
Section  B.3).  The  symbiotic  relationship  between  my  neuroscience  and  information  theory 
interests  came  full  circle  when  the  brain-machine  interface  paradigm  engaged  my  curiosity  to 
understand  posterior  matching  feedback  information-theoretic  schemes  from  control  viewpoints. 

The  essence  of  the  optimal  feedback-based  posterior  matching  scheme  -  to  communicate  to 
the  decoder  information  about  the  message  that  is  “statistically  independent  of  everything  the 
decoder  has  seen  so  far”  -  led  to  my  hypothesis  that  its  structure  is  intimately  related  to 
stochastic  control.  Indeed,  I  was  able  to  demonstrate  that  it  indeed  is  acting  like  an  optimal 
stochastic  controller,  over  the  space  of  posteriors,  or  decoder’s  beliefs  about  the  message, 
acting  to  geometrically  “steer”  the  decoder’s  belief  towards  certainty  at  the  message  point.  It 
turns  out  that  a  simple  reward  function  in  this  setting  pertains  to  the  “reduction  in  distance 
towards  certainty”,  and  this  defines  a  canonical  stochastic  control  problem  with  the  PM  scheme 
as  an  optimal  solution.  Moreover,  a  Lyapunov  function  exists,  whose  structure  encodes  how 
fast  error  probabilities  tend  to  0  [5], 

I  further  became  intrigued  about  dynamical  systems  interpretations  of  the  posterior  matching 
encoding  scheme  -  which  has  an  iterated  function  system  structure  with  a  positive  Lyapunov 
exponent.  This  observation  leads  to  a  reverse  iterated  function  type  of  decoder,  with  a  negative 
Lyapunov  exponent  equal  to  the  negative  mutual  information.  These  observations  provide  an 
alternative,  dynamical  systems-based  proof  that  the  posterior  matching  encoding  scheme,  wed 
with  this  dual  decoder,  achieves  capacity.  Further  analysis  of  this  scheme,  and  understanding 
dualities  between  channel  coding  with  feedback  and  source  coding  with  feedforward,  has  led  to 
an  optimal,  low-complexity  recursive  scheme  for  source  coding  with  feedforward  causal  side 
information  [6], 

By  understanding  the  nature  of  optimal  communication  with  feedback,  and  considering  timing 
channels  -  which  are  channels  that  have  state  and  internal  feedback  -  I  have  become  generally 
interested  in  high-level  principles  that  underlie  optimal  structures  of  sequential,  causal, 
information  theoretic  problems  with  feedback.  This  has  led  to  a  focus  on  a  very  general 
formulation  of  such  problems  and  the  interplay  between  information  theory,  control  theory,  and 
thermodynamics.  This  approach  has  been  able  to  unify  a  collection  of  seemingly  disparate 


results  in  the  separate  communities.  With  this,  we  have  a  general  proof  of  a  large  variety  of 
results  composing  information  theory  of  timing  channels,  capacity  of  certain  channels  with 
internal  state  and  feedback,  rate-distortion  theorems,  fundamental  limits  of  decentralized 
control,  with  practical  implications  for  the  design  of  brain-machine  interfaces  [7], 
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Many  current  viewpoints  about  how  neural  processes  integrate  to  elicit  complex  brain  function 
posit  that  populations  of  neurons  in  the  human  brain  are  connected  to  form  functionally 
specialized  assemblies.  With  the  increasing  ability  to  record  multiple  neural  signals  at  different 
brain  areas  simultaneously,  one  core  issue  in  neuroscience  research  isunderstanding  the 
mechanistic  phenomena  and  how  to  analyze  these  ensemble  recordings  and  infer  the  structure 
of  these  mechanisms.  One  such  approach  to  attempt  to  understand  this  mechanistic 
phenomena  is  from  a  causal  network  viewpoint. 

The  directed  information  -  independently  developed  by  information  theorists  Massey  and 
Rissanen,  is  an  information-theoretic  quantity  analogous  to  mutual  information  that  encodes  the 
fundamental  limits  of  communication  with  feedback.  From  the  viewpoint  of  a  log  likelihood  ratio, 
it  can  be  shown  to  be  philosophically  consistent  with  “Granger  causality”,  in  that  it  measures 
directionality  of  causality  (e.g.,  X  causing  Y)  by  assessing  whether  or  not  past  values  of  X  and  Y 
help  to  predict  the  future  of  Y  better  than  only  past  values  of  Y  [8],  Remarkably,  the  directed 
information  has  been  used  very  sparingly  on  simultaneously  recorded  data  sets  to  answer 
questions  of  causality  -  despite  the  growing  importance  of  these  questions  in  a  variety  of  fields, 
including  economics,  computer  security,  systems  biology,  and  neuroscience.  In  [9]  we 
demonstrate  provably  good  directed  information  estimation  algorithms.  We  have  further 
attempted  to  develop  parsimonious  approaches  approximations  to  infer  causal  networks  with 
tree-like  structures,  and  have  developed  a  provably  good  methodology  to  infer  causal  directed 
dependence  trees  that  can  be  efficiently  implemented  [10].  Because  the  brain  is  inherently 
dynamic  and  time-varying,  we  are  tailoring  these  algorithms  to  understand  time-varying  causal 
influences  that  are  efficiently  solvable  and  provably  good  [10]. 
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